The present invention relates to a high-speed vision sensor having an image processing function.
High-speed image processing is required to operate high-speed robots in a factory automation (FA) system or the like. For example, when a robot is configured with a feedback loop between its vision sensor and its actuator, the actuator can be controlled in units of milliseconds. Accordingly, the image processing speed matching this speed is required. However, because the image processing speed in modern vision systems is limited by the video frame rate, the robot can only operate at low speeds matching the image processing speed. It is impossible to take full advantage of the robot""s performing capability.
There are some high-speed CCD cameras that can take images at intervals of about one (1) millisecond. In these devices, the images are temporarily stored in memory and later read and processed. Therefore, the devices can be applied to such applications as image analysis. However, the devices have almost no real-time capability, and are not suitable for controlling robots or similar real-time applications.
In order to overcome these problems, institutes such as the Massachusetts Institute of Technology, the California Institute of Technology, and Mitsubishi Electric Corporation have been researching a vision chip that combines the image reading portion and the processing portion into one unit (xe2x80x9cAn Object Position and Orientation IC with Embedded Imager,xe2x80x9d David L. Standley, Solid State Circuits, Vol. 26, No. 12, Dec. 1991, pp. 1853-1859, IEEE); xe2x80x9cComputing Motion Using Analog and Binary Resistive Networks,xe2x80x9d James Hutchinson, et al., Computer, Vol. 21, March 1988, pp. 52-64, IEEE); and xe2x80x9cArtificial Retinasxe2x80x94fast versatile image processors,xe2x80x9d Kazuo Kyuma, et al., Nature, Vol. 372, Nov. 10, 1994). However, these chips employ a fixed analog circuit that is easy to integrate. Accordingly, these circuits have such shortcomings as requiring subsequent-processing of output signals and a lack of universality. Hence, the type of image processing they can perform is limited to special applications.
Japanese Unexamined Patent Application Publication HEI-10-145680 has proposed a vision chip that is capable of performing universal image processing. This vision chip is provided with a processing element for each photodetector. An analog-to-digital converter is provided for each photodetector row. Therefore, the vision chip can reduce the processing time through parallel processing. The vision chip can also reduce the number of transmission lines between the photodetectors and the processing elements, achieving an optimal integration level for both.
However, since this vision chip is configured to use the processing elements themselves when transferring data thereto, processing cannot be performed during image transfers. It is noted that it is necessary to perform image processing over a plurality of images in order to detect the shape of objects or to detect movement. Since conventional methods temporarily read a plurality of images into memory before processing, such processing cannot be performed in real-time.
In view of the foregoing, it is an object of the present invention to provide a multi-pixel high-speed vision sensor which has a simple circuit construction and which is capable of performing efficient high-speed calculations even over a plurality of images.
In order to attain the object, the high-speed vision sensor of the present invention comprises: at least one photodetector array, each having a plurality of photodetectors which are arranged two-dimensionally in a plurality of rows and in a plurality of columns; an analog-to-digital converter array having a plurality of analog-to-digital converters which are arranged one-dimensionally such that each analog-to-digital converter corresponds to one row in the at least one photodetector array, each analog-to-digital converter converting, into digital signals, analog signals which are successively outputted from the photodetectors in the corresponding row; a parallel processing system, including a parallel processing element array and a shift register array, the parallel processing clement array having a plurality of processing elements which are arranged two-dimensionally in a plurality of rows and in a plurality of columns and in one-to-one correspondence with the plurality of photodetectors in the at least one photodetector array, each processing element performing a predetermined calculation on digital signals transferred from the analog-to-digital converter array, the shift register array having a plurality of shift registers which are disposed in one-to-one correspondence with the plurality of analog-to-digital converters and in one-to-one correspondence with the plurality of rows of processing elements, each shift register successively transferring digital signals, which are received from the corresponding analog-to-digital converter and which are equivalent to signals outputted from the photodetectors in a corresponding photodetector row, to predetermined processing elements in the corresponding row; and a control circuit controlling the photodetector array and the analog-to-digital converter array to output digital signals for a single frame and controlling the shift register array to transfer the digital signals of the single frame to the parallel processing element array, and thereafter controlling the photodetector array and the analog-to-digital converter array to output digital signals for the next frame and controlling the shift register array to transfer the digital signals of the next frame to the parallel processing element array, while simultaneously controlling the parallel processing element array to perform the predetermined calculation onto the single frame.
According to the present invention, the plurality of processing elements are provided in one to one correspondence with the plurality of photodetectors. It is therefore possible to perform high-speed image processing through parallel processing. Additionally, the shift registers are used as being dedicated to transferring data for processing. Accordingly, image processing can be performed during the transfer process. Image processing can be performed efficiently by reducing the wait time for both of the transfer process and the calculating process, thereby reducing the overall processing time. It is therefore possible to achieve a pipeline operation. It is possible to perform high-speed image processing, and particularly real-time processing. An analog-to-digital converter is provided for each row. Accordingly, the total number of transmission paths can be reduced.
When the high-speed vision sensor has a plurality of photodetector arrays, the parallel processing system may preferably include, in correspondence with each processing element row, a plurality of lines of shift registers, the number of the plurality of lines being equal to the number of the plurality of photodetector arrays.
With this construction, the plurality of lines of shift registers are provided for transferring, to each row of processing elements, data from the corresponding row of photodetectors in the plurality of photodetector arrays. Accordingly, even when the high-speed vision sensor has plural photodetector arrays, image signals from the plural photodetector arrays can be transferred independently from one another, thereby requiring no extra time for the transfer process than when the high-speed vision sensor has only one photodetector array. Therefore, the image processes, including the calculating processes, can be performed at a high rate of speed.
The plurality of photodetector arrays may be disposed at positions different from one another. The control circuit may include: a parallel processing control portion controlling the respective lines of shift registers to transfer images which are taken at different positions and outputted from the corresponding photodetector arrays, and controlling the parallel processing system to perform calculations, based on image signals obtained by the plurality of photodetector arrays, to determine the amount of positional shift, of an object, between its images taken by the plurality of photodetector arrays; and a calculating portion calculating three-dimensional positional information of the object based on the determined amount of positional shift and information on the position of each photodetector array and on the direction in which each photodetector array takes images.
By arranging the plurality of photodetector arrays in different positions and by controlling the photodetector arrays to take images of the same object, images obtained by the respective photodetector arrays will correspond to images of the object projected in predetermined directions. It is possible to determine the three-dimensional position of the object by using trigonometry based on the positions of the same portion of the object in the respective images. The above-described process requires at least two photodetector arrays. However, three or more photodetector arrays can be used in order to improve accuracy and to reduce blind spots.
In order to determine the three-dimensional position of the object, the plurality of photodetector arrays may preferably be arranged such that data transfer directions, along which the rows of the photodetectors in the respective photodetector arrays extend to transfer data from the photodetectors, are lined up in the same direction and such that corresponding photodetectors in the plural photodetector arrays are positioned to be shifted from one another in the data transfer direction. The parallel processing control portion may control the parallel processing system to calculate, based on the image signals obtained by the plural photodetector arrays, the amount of positional shift along the data transfer direction among the images taken by the plural photodetector arrays.
In this way, the direction of data transfer is the same for all the photodetector arrays, and corresponding photodetectors in the photodetector arrays are disposed in positions which are shifted from one another in the data transfer direction. Accordingly, a matching point of the images detected by the photodetector arrays will appear in different positions in the images taken by the plural photodetector arrays. This shift in position will occur in the same direction as that in which the photodetector arrays are shifted, that is, the direction of data transfer. Because the direction in which the shift in position occurs is the same as the data transfer direction, it becomes easy to shift one image in relation to another image by controlling the transfer amount of data transfer. It therefore becomes easy to search the same part of images and, therefore, to detect three-dimensional positional information.
Preferably, the high-speed vision sensor may further comprise a pixel shift amount storage device storing the amount of positional shift, along the data transfer direction, which is calculated by the parallel processing system with respect to the plurality of images taken by the plurality of photodetector arrays. The parallel processing control portion may control, based on the stored positional shift amount, the transfer position of the processing elements, to which each line of shift registers transfers an image signal.
With this construction, it is possible to perform calculations using, as a reference, the positional shift amount calculated for the previous frame. Hence, calculations of the positional shift amount and recognition of the matching point can be performed more quickly.
Alternatively, the high-speed vision sensor may further comprise: a timing controller that controls the plurality of photodetector arrays to take images at timings independent from one another; and a beam splitter that enables the plurality of photodetector arrays to pick up images from the same direction. The control circuit may include a parallel processing control portion controlling the plural lines of shift registers to transfer images taken by the respective photodetector arrays at the different times, and controlling the processing elements to perform calculation onto the images taken at the different times.
The timings when images are taken by the photodetector arrays are controlled independently from one another. Thus, the times when images are obtained are controlled. Additionally, the photodetector arrays are made to take images of the object from the same direction. Accordingly, the plural photodetector arrays can take images at different times at short intervals. It is possible to obtain a plurality of frame images at different times by transferring those images using the different lines of shift registers. By performing calculations onto those images, it is possible to easily attain the recognition of a high-speed moving object or of movement.
The high-speed vision sensor may further comprise a filtering/beam splitter mechanism enabling the plurality of photodetector arrays to pick up, from the same direction, color-separated images, of an object, which have colors different from one another. The control circuit may include a parallel processing control portion controlling the plural lines of shift registers to transfer the color-separated images outputted from the respective photodetector arrays and controlling the processing elements to perform calculations onto the color-separated images.
With this construction, color-separated image data is transferred from each photodetector array to the processing elements. Hence, the processing elements can easily perform image processes such as color adjustment or color correction.
The high-speed vision sensor may further comprise data supply mechanism supplying predetermined data required for image processing calculation. Each shift register may successively transfer output signals, outputted from the corresponding analog-to-digital converter, and the predetermined data, supplied from the data supply mechanism, to the predetermined processing elements in the corresponding processing element row. The control circuit may control the photodetector array, the analog-to-digital converter array, the parallel processing system, and the data supply mechanism.
With this construction, not only image data from the photodetectors, but also various types of data, necessary for processing, is transferred to the processing elements using the dedicated shift registers. Accordingly, it is possible to perform these data transfer operation in parallel with the image processing operation.
The data supply mechanism may preferably include a time-sharing mixing unit combining, according to a time-sharing manner, output signals outputted from the analog-to-digital converters and the predetermined data. By combining, in a time-sharing manner, image data and the predetermined data necessary for processing, both types of data can be transferred reliably and efficiently on the same line of shift registers.
Alternatively, the parallel processing system may further include a plurality of data-transfer shift registers which are arranged in one-to-one correspondence with the plurality of rows of processing elements, each data-transfer shift register supplying the predetermined data to the respective processing elements in the corresponding row.
With this construction, the high-speed vision sensor is provided with two separated groups of shift registers: one group being for transferring input image data; and the other being for transferring the predetermined data required for image processing. Accordingly, the high-speed vision sensor can achieve a pipeline process by performing in parallel: the calculation operation with the processing elements, the transferring operation of input image data, and the transferring operation of other data required for image processing, thereby enabling the high-speed image processing and, particularly, the real time image processing.
When the high-speed vision sensor has a plurality of photodetector arrays, the parallel processing system may preferably include a single line of shift register for each processing element row, the single line of shift register being used in a time-sharing manner to transfer the output signals from the plurality of photodetector arrays.